Autonomous diagnosis of ear diseases from biomarker data

ABSTRACT

A fully autonomous system is used to diagnose an ear infection in a patient. For example, a processor receives patient data about a patient, the patient data comprising at least one of: patient history from medical records for the patient, one or more vitals measurements of the patient, and answers from the patient about the patient&#39;s condition. The processor receives a set of biomarker features extracted from measurement data taken from an ear of the patient. The processor synthesizes the patient data and the biomarker features into input data, and applies the synthesized input data to a trained diagnostic model, the diagnostic model comprising a machine learning model configured to output a probability-based diagnosis of an ear infection from the synthesized input data. The processor outputs the determined diagnosis from the diagnostic model. A service may then determine a therapy for the patient based on the determined diagnosis.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/713,283, filed Aug. 1, 2018, the disclosure of which is herebyincorporated by reference herein in its entirety.

BACKGROUND

This invention relates generally to autonomously assessing, diagnosing,and prescribing therapy for ear infections, such as acute otitis media(AOM). To diagnose ear infections, physicians typically utilize anotoscope, pneumatic otoscope or tympanometry as a tool. However, thesetechniques are not infallible and are subject to error, thus resultingin a historical over-diagnosis of ear infections. Problematically,variance in physician analysis of the results of use of otoscopy,pneumatic otoscopy or tympanometry is a cause in this historicalover-diagnosis. Existing systems lack an objective tool for ensuring anaccurate diagnosis and require expert physician interpretation, lackingthe capability of a fully-autonomous tool to automatically diagnosis earinfections and prescribe antibiotics, which would reduce a cost burdenon society caused by the over-diagnosis brought on by the existingsystems.

SUMMARY

Systems and methods are described herein for using an objective servicefor accurately assessing, diagnosing, and prescribing therapy for earinfections. In an embodiment, a machine learning model generates anoutput of a probability-based diagnosis based on inputs includingbiomarker features and patient history. The received biomarker featuresmay include overlapping and/or non-overlapping biomarker featurefeatures. For example, the biomarker features may be image biomarkersthat indicate one or more anatomical features found in an image of anear, as well as a location at which the one or more anatomical elementsare present in the ear. The biomarker features may also include acousticbiomarkers that are derived from responses of an ear to a pressurestimulus. The probability-based diagnosis may indicate a diagnosis alongwith a probability that the diagnosis is correct, which may be used, inconjunction with rules or probability-based models, to output adefinitive diagnosis to a user (e.g., a doctor or other health careprofessional, or to the patient).

In an embodiment, a processor receives patient data about a patient,where the patient data includes patient history from medical records forthe patient, one or more vitals measurements of the patient, and/oranswers from the patient about the patient's condition. As used herein,answers from the patient may be obtained directly from the patient orindirectly, e.g., from the patient's proxy. The patient's condition mayinclude a present condition and/or a medical history of the patient,including a history of the present illness for the patient. Theprocessor receives a set of biomarker features extracted frommeasurement data taken from one or more ears of the patient. Theprocessor synthesizes the patient data and the biomarker features intoinput data, and applies the synthesized input data to a traineddiagnostic model. In some embodiments, the diagnostic model comprises amachine learning model configured to output a probability-baseddiagnosis of an ear infection from the synthesized input data. Theprocessor then outputs the determined diagnosis from the diagnosticmodel. A service (e.g., a clinical decision support system) may thendetermine a therapy for the patient based on the determined diagnosis,for example, using an expert system that applies a set of rules fordetermining the therapy.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustrative block diagram of components used in a systemfor autonomously diagnosing an ear disease and generating a therapytherefor, and includes an illustrative flow of data through the system.

FIG. 2 is an illustrative diagram of modules of an image analysisservice used to produce image biomarkers used in the diagnosis of an eardisease.

FIG. 3 is an illustrative diagram of an acoustic response analysisservice used to produce acoustic biomarkers used in the diagnosis of anear disease.

FIG. 4 is an illustrative diagram of a classifier service used toautonomously generate a probability-based diagnosis of an ear disease.

FIG. 5 is an illustrative diagram of a therapy determination serviceused to autonomously generate a therapy for output to a user.

The figures depict various embodiments of the present invention forpurposes of illustration only. One skilled in the art will readilyrecognize from the following discussion that alternative embodiments ofthe structures and methods illustrated herein may be employed withoutdeparting from the principles of the invention described herein.

DETAILED DESCRIPTION Overview

FIG. 1 is an illustrative block diagram of components used in a systemfor autonomously diagnosing an ear disease and generating a therapytherefor, and includes an illustrative flow of data through the system.FIG. 1 depicts system 100, which includes patient file server 110, imagecapture analysis 120, image analysis service 130, acoustic responsecapture apparatus 140, acoustic response analysis service 150,classifier service 160, and therapy determination service 170. Whiledepicted as separate components of system 100, some or all of imageanalysis service 130, acoustic response service 150, classifier service160, and therapy determination service 170, may be consolidated into oneserver or apparatus. For example, such a consolidated apparatus may be ahardware module and/or software module used by individual doctors forassistance in diagnosing ear disease, such as AOM. Further, eachdepicted service may be implemented in one or more servers, or may beinstantiated at a client site (e.g., a doctor's office) where a client(e.g., doctor) subscribes to the service. Alternatively, the service maybe implemented on a client device (e.g., a patient's mobile device) incommunication with one or more servers.

Patient file server 110 may include a database that stores electronicfiles of patients. The term patient, as referred to herein, may be anyhuman being, such as a child, adolescent, or adult. The electronic filesof each patient may indicate the long-term history of the patient. Theterm long-term history, as used herein, refers to historical datagathered by one or more doctors, clinicians (e.g., doctor's assistants,nurses, etc.), or input by the patient himself or herself, or proxy suchas parent or guardian. The historical data includes health data. Healthdata may include temperature data. (e.g., last measured temperaturealong with a time and date at which the temperature was taken, orseveral patient temperatures each with their respective times anddates). Temperature data may be accompanied by the technique by whichthe temperature was measured (e.g., oral, anal, forehead, device). Thehealth data may also include any other parameters corresponding to thepatient, such as history of present illness, blood pressure, age,gender, weight, height, current prescriptions, pain severity assessment(e.g., on a scale from 1 to 10), duration of symptoms, allergies,frequency of ear issues, and the like. Each piece of historical data maybe accompanied by time and/or date information.

In an embodiment, when current patient state is received by an operatorduring a visit by a patient, the current patient state may bedocumented, but be immediately sent to an electronic patient file. Theterm operator, as used herein, is likely not to be a medically-trainedperson, but may be a clinician. The systems and methods describedherein, for example, may result in a therapy determination without aneed to consult a clinician. The term current patient state, as usedherein, refers to health data of a patient that is current as to a timeat which a diagnosis is taken. For example, if a patient visits adoctor's office on Jul. 24, 2019 at 5:00 pm, the current patient statemay include temperature, height, and weight data of the patient on Jul.24, 2019 at, or around, 5:00 pm. Where the current patient state isimmediately sent to the electronic patient file, the electronic patientfile is complete. However, for example where the current patient stateis not immediately sent to the electronic file or the electronic patientfile is otherwise incomplete, and an operator may, in having classifierservice 160 output a diagnosis, separately transmit or input the currentpatient state to classifier service 160. For example, the operator mayinput into a user interface of an application corresponding toclassifier 160 the current patient state. The therapy determinationservice 170 may also prompt the operator and/or the patient withquestions that are used to determine the current patient state.

Image capture apparatus 120 may be any apparatus used to capture animage of a patient's ear (e.g., the inner ear of the patient). Imagecapture apparatus 120 may, for example, be an otoscope. Image captureapparatus 120 may perform visible light imaging. Image capture apparatus120 may perform optical coherence tomography (OCT) imaging. In anembodiment, image capture apparatus 120 captures an image of the eardrumand/or adjacent structures to the eardrum. Image capture apparatus 120may have snapshot OCT capability and/or continuous A-scan capability(e.g., where focus of image capture apparatus 120 is continuouslyadjusted as different images (e.g., OCT images) are taken). Imagecapture apparatus 120 may have a plurality of different centerwavelengths for OCT imaging. Image capture apparatus 120 may be capableof producing two-dimensional (2D) and/or three-dimensional (3D) images,along with timestamps of when those images were taken.

Image capture apparatus 120 may navigate to a field of view includingpredetermined anatomical structures, such as an ear drum. The navigationmay be performed automatically based on instructions from a processor,or may be performed manually by a user, such as a doctor doctor'sassistant, or an untrained user. In an embodiment, image captureapparatus 120 includes, or is operably coupled to, a display. Thedisplay may indicate feedback from image analysis service 130, and mayindicate an enlarged view of images captured by a camera sensor of imagecapture apparatus 120 of the inside of an ear.

Image analysis service 130 receives images from image capture apparatus120 and processes the images. Where the images are sufficient foridentifying image biomarkers, image analysis service 130 extracts theimage biomarkers and transmits the image biomarkers to classifierservice 160. Where the images are not sufficient for identifying imagebiomarkers, feedback is provided to at least one of image captureapparatus 120 and an operator of image capture apparatus 120.Determination of whether images are sufficient for identifying imagebiomarkers, extraction of image biomarkers, and provision of feedback isdescribed in further detail below with respect to FIG. 2. The term imagebiomarker, as used herein, refers to a data structure that indicates ananatomical feature (e.g., a blood vessel, a fissure, etc.) found in animage or a characteristic disease feature (e.g., otalgia, magnitude oftympanic membrane bulging). The biomarkers may also include, forexample, the severity of drum hyperemia, the presence of effusion behindan eardrum, and/or the reflectivity of the effusion at a plurality ofcentral wavelengths, and/or an amount of convexity or concavity of aneardrum. The image biomarker data structure may include additionalinformation, such as a location in the image in which the anatomicalfeature was found, a location in the ear corresponding to where theanatomical feature sits, a confidence value indicating a probabilitythat the anatomical feature actually is what was identified, and thelike. In an embodiment, instead of or in addition to indicating theprobability, the image biomarker data structure includes a determinedpresence of a known abnormality that may interfere with diseaseassessment, such as debris, imaging artifacts (e.g., reflections,shadow, motion), implants, etc. In an embodiment, image analysis serviceperforms a sufficiency analysis in real time, or in substantially realtime, to continuously provide feedback to the operator to aid in imageacquisition.

Acoustic response capture apparatus 140 captures an acoustic response ofan ear of a patient to one or more stimuli and transmits the acousticresponse to acoustic response analysis service 150. Acoustic responsecapture apparatus 140 may be any device that applies pressure to ananatomical feature of a patient's ear (e.g., an eardrum). For example,acoustic response capture apparatus 140 may be a device that performspneumatic otoscopy or tympanometry. In an embodiment, acoustic responsecapture apparatus 140 captures an acoustic response from standardtympanometry. In standard tympanometry, the ear canal is pressurized byacoustic response capture apparatus with air over a spectrum ofpressures while sound waves at a specific frequency are transmitted tothe ear. The based on the pressurization, data is received that, ifgraphed, plots ear pressure against at least one of absorbed sound andcompliance of the eardrum. The data, or graphs of the data, are comparedto template data, to determine which of a plurality of templates thedata or graphs match. For example, data may be determined to match anormal response, a response indicating that fluid is present, a responseindicating negative pressure exists in the middle ear, and the like.

In an embodiment, acoustic response capture apparatus 140 captures anacoustic response from wideband tympanometry. Wideband tympanometryworks in a similar manner to standard tympanometry, except that, whilestandard tympanometry typically uses one or a small number offrequencies, wideband tympanometry uses a large number of frequencies(e.g., over 100 frequencies). Because different materials absorbdifferent frequencies, wideband tympanometry may provide more robustdata than standard tympanometry. For example, wideband tympanometry maybe used to, beyond detecting middle ear fluid (due to data matching amiddle ear fluid template), also detect a makeup of the fluid (e.g.,whether the fluid is or is not infected). The data output by acousticresponse capture apparatus 140 when using wideband tympanometry can befed into apparatus 150 which may use machine learning models to quantifythe presence of acoustic biomarkers for input to apparatus 160.

Other forms of tympanometry may be used as well by acoustic responsecapture apparatus 140, such as wideband tympanometry at ambientpressure, where wideband tympanometry is used at many frequencies, butonly at ambient pressure, thus alleviating the need to pressurize theear canal. The data output by acoustic response capture apparatus 140when using wideband tympanometry can be fed into apparatus 150 which mayuse machine learning models to quantify the presence of acousticbiomarkers for input to apparatus 160.

Acoustic response analysis service 150 receives acoustic responses fromacoustic response capture apparatus 140 and extracts acoustic biomarkersfrom the acoustic responses. The term acoustic biomarkers, as usedherein, refers to output from a machine learning model that may be usedto classify a disease. For example, the data and/or graphs output byacoustic response capture apparatus may be input into a machine learningmodel. The output may be indicia including the presence or amount offluid, the classification of mucoid versus serous fluid, the type oftympanometry classification (e.g., normal, fluid present, negativepressure in middle ear, etc.), a prediction of the presence of diseaseitself, and/or the detection of the presence of an obstruction in theear. Any or all of these outputs may form acoustic biomarkers. In anembodiment, the acoustic responses themselves may be used as acousticbiomarkers, without input through a machine learning model to extractbiomarkers, where a model at classifier service 160 directly translatesthese acoustic biomarkers into a disease classification. Acousticbiomarkers and image biomarkers may be collectively referred to hereinas biomarkers. Further details of acoustic response analysis service 150are described in further detail below with respect to FIG. 3.

Classifier service 160 receives patient history from patient file server110. Classifier service 160 also receives image biomarkers from imageanalysis service 130 and/or acoustic biomarkers from acoustic responseanalysis service 150. While FIG. 1 depicts receipt of both imagebiomarkers and acoustic biomarkers, in an embodiment, only one or theother of image biomarkers and acoustic biomarkers is received byclassifier service 160. Classifier service 160 may also receive currentpatient state data. While FIG. 1 depicts classifier service 160 asreceiving current patient state data from patient file server 110,classifier service 160 may receive the current patient state data fromanother source, such as direct input from a doctor or doctor'sassistant, from the patient himself or herself, or from the operator.

Classifier service 160 synthesizes the received data and feeds thesynthesized data as input into a machine learning model. The machinelearning model outputs a probability-based diagnosis, which classifierservice 160 transmits to therapy determination service 170. The termprobability-based diagnosis refers to a diagnosis as well as aprobability that the diagnosis is accurate. For example, the diagnosismay be a probability that a patient does, or does not, have AOM. In anembodiment, a diagnosis is sent without a corresponding probability thatthe diagnosis is accurate. Further details of classifier 160 aredescribed in further detail below with respect to FIG. 4.

Therapy determination service 170 receives the probability-baseddiagnosis and generates a therapy for the patient. The therapydetermination service 170 takes as input the disease diagnosis fromapparatus 160 as well as patient information from apparatus 110. Service170 may be a Clinical Decision Support System, and may be implemented asa rule-based system or as a machine learning system. As an example, thetherapy decision may be to prescribe a prescription drug, such as anantibiotic. The therapy may alternatively be watchful observation withfollow-up, or referral to a specialist. Therapy determination service170, and therapy determination generally, is described in further detailbelow with respect to FIG. 5.

FIG. 2 is an illustrative diagram of modules of an image analysisservice used to produce image biomarkers used in the diagnosis of an eardisease. Image analysis service 230 is depicted as including sufficiencydetermination module 231, feedback module 232, image biomarkerextraction module 233, sufficiency parameters database 234, and imagebiomarker attributes database 235. The modules depicted in FIG. 2 may beexecuted by a processor of a server or device hosting image analysisservice 230. The execution of the modules may be driven bycomputer-readable instructions on a non-transitory computer-readablemedium that, when executed, cause the processor to perform theoperations described with respect to FIG. 2. The databases depicted inFIG. 2 may be local to a device or server hosting image analysis service230, or may be remote to such a device or server, and accessible by wayof a network, such as a local area network or the Internet. Thesufficiency parameters database 234 and image biomarker attributesdatabase 235 may be the learned weights of a machine learning model usedto calculate outputs of the image analysis service 130.

Sufficiency determination module 231 determines whether an imagereceived from image capture apparatus 120 is sufficient for extractionof image biomarkers. Sufficiency determination may be based quality(such as the input image/signal quality) or protocol (such as whetherrequired anatomical landmarks are present in the image). Sufficiencydetermination module 231 may determine sufficiency of an image based onparameters defined in sufficiency parameters database 234. For example,sufficiency parameters database 234 may include image quality parametersor the learned weights of one or more machine learning models used tocalculate image quality. The quality parameters may be based onresolution, a lack of debris in the photograph, or the learned weightsof a machine learning model.

The quality parameters may be driven by a deep learning network that,from a training set, models aspects of an image or a transformed image,that are of sufficient quality, and measures received images against anoutput of the model. For example, the model may be trained to determinewhether an image includes a correct field of view, whether the image hasa background that is set to a constant value, whether the image isdown-sampled, etc. Using the model, or other quality parametersindicated by sufficiency parameters database 234, sufficiencydetermination module 231 determines whether received images are ofsufficient quality. In an embodiment where multiple images are input toimage analysis service 130 by way of a video captured by image captureapparatus 120, sufficiency determination module 231 may use an RNN(recurrent neural network) model for video frame selection for imagequality. Because the data is being processed on a continuous basis, theRNN may be used to select a plurality of consecutive sufficient framesto avoid being overburdened with processing all frames of the video,where the consecutive sufficient frames are, alone, sufficient for imageanalysis. Machine learning models, as described herein, may be any formof machine model, such as convolutional neural networks (CNNs), RNNs,and the like.

Sufficiency parameters database 234 may also include image adherenceparameters. For example, sufficiency parameters database 234 may includeentries describing various landmarks and required positions of thoselandmarks within a viewing frame. For instance, an entry may indicatethat a tympanic membrane must be within a field of view. Sufficiencydetermination module 231 may process an image to determine whether theimage protocol adherence requirements in sufficiency parameters database234 are met when determining whether the image is sufficient forextraction of image biomarkers. As an example, sufficiency determinationmodule 231 may obtain a set of samples of a portion of an image, whereeach sample corresponds to a location in the ear (e.g., ear canal,tympanic membrane, malleus, umbo, light reflex, etc.). The samples maybe applied by sufficiency determination module 231 to a trained featuredetection model (which may also be stored in sufficiency parametersdatabase 234), the model comprising a neural network that is configuredto output a likelihood of whether the sample contains an ear imageobject. Sufficiency determination module 231 may determine, from theoutput, whether the image sufficiently adheres to the image adherenceparameters.

Where an image is determined to be insufficient for extraction of imagebiomarkers, feedback module 232 transmits feedback to image captureapparatus 120. In an embodiment where image capture apparatus 120 has adisplay, feedback module 232 transmits to image capture apparatus 120 anindication, for inclusion on the display, of why an image isinsufficient. For example, if the quality is a problem, a color scale(e.g., red, yellow, green) may indicate the degree to which the qualitywas insufficient. If image adherence is a problem, feedback module 232may transmit instructions (e.g., an arrow on the display) to move theview of the image capture apparatus 120 to one side or the other.Feedback module 232 may cause the display to circle an area of an imagethat needs to be centered, and may instruct the operator to center thatportion of the image.

Feedback module 232 may transmit any sort of instruction for moving afield of view of a camera sensor in any direction, such as forward orbackward (to improve depth of view), left, right, up, down, ordiagonally (to improve the centering of an anatomical feature orstructure), etc. These instructions may be displayed on a display, maybe output through a speaker, may be printed to text, etc. In anembodiment, image capture apparatus 120 may receive instructions fromimage analysis service 130 to automatically adjust the position of acamera sensor in the same manners described above with respect to manualmovement. In an embodiment, image capture apparatus 120 may receive acommand to output, on the display, a notification that sufficientquality images have received, and thus image capture may be stopped.While the term image is used herein, image capture apparatus 120 maycapture video, from which individual images are derived.

In an embodiment, feedback module 232 may command image captureapparatus 120 (or a peripheral operably coupled thereto) to outputauditory signals to indicate needed changes to image quality oradherence, or to indicate that. For example, image analysis service 130may command image capture apparatus 120 to beep or otherwise sound whensufficient quality imaging has been received, and thus imaging can bestopped. Similarly, feedback module 232 may command image captureapparatus 120 (or a peripheral operably coupled thereto) to generatehaptic output to alert an operator to any alert or notificationdescribed herein (e.g., a vibration to indicate that sufficient exam hasbeen acquired and thus further imaging is no longer needed).

Feedback module 232 may use any mechanism described above to direct theoperator to an optimal imaging location (e.g., in the external meatus).The optimal imaging location may be with respect to anatomical features,location of debris, pathology of the eardrum, and the like, asdetermined, e.g., by a machine learning model. In an embodiment,feedback module 232 may instruct the operator to clean the device orclear debris (e.g., cerumen) near an imaging location.

Image biomarker extraction module 233 extracts image biomarkers from theimages received from image capture apparatus 120. Image biomarkersinclude the detection of relevant anatomy and disease/pathologicalfeatures associated with the diagnosis. Image biomarker extractionmodule 233 may determine what attributes of an image form a biomarkerbased on entries of image biomarker attributes database 235. Imagebiomarker attributes database 235 includes entries that indicatepatterns associated with anatomical features, such as fissure patterns,bulging patterns, darkening and lightening patterns relative to abackground, and the like. Image biomarker attributes database 235 mayinclude one or more trained machine models that may be used by imagebiomarker extraction module to input an image through, and receive asoutput a determination of biomarkers. Image biomarker extraction module232 extracts, from the image, the location of the biomarker (e.g., withreference to an anatomical structure, such as a tympanic membrane oreardrum). The location may be indicated in terms of coordinates that maybe normalized relative to an anatomical structure. Image biomarkerextraction module 232 includes, in the image biomarker, the location, aswell as an indication of what the biomarker is (e.g., a fissure). In anembodiment, image biomarker extraction module 233 may also include aconfidence value, or probability, that the biomarker actually is whatthe image biomarker indicates it is (e.g., an 80% confidence that thisis a fissure). Image analysis service 230 transmits the image biomarkerto classifier service 160.

Further details of how to detect anatomical features in images andextract biomarkers from images are described (using terms relating toextracting “features”) in U.S. Pat. No. 10,115,194, issued Oct. 30,2018, the disclosure of which is hereby incorporated by reference hereinin its entirety.

FIG. 3 is an illustrative diagram of an acoustic response analysisservice used to produce acoustic biomarkers used in the diagnosis of anear disease. Acoustic response analysis service 350 includes acousticbiomarker extraction module 351 and acoustic biomarker attributesdatabase 352. The modules depicted in FIG. 3 may be executed by aprocessor of a server or device hosting acoustic response analysisservice 350. The execution of the modules may be driven bycomputer-readable instructions on a non-transitory computer-readablemedium that, when executed, cause the processor to perform theoperations described with respect to FIG. 3. The databases depicted inFIG. 3 may be local to a device or server hosting acoustic responseanalysis service 330, or may be remote to such a device or server, andaccessible by way of a network, such as a local area network or theInternet. Acoustic biomarker extraction module 351 may execute a machinelearning model to translate outputs from acoustic response captureapparatus 140. The machine learning model may be retrieved from acousticbiomarker attributes database 352. Further details of biomarkerextraction and use of the machine learning model are described abovewith respect to FIG. 1.

FIG. 4 is an illustrative diagram of a classifier service used toautonomously generate a probability-based diagnosis of an ear disease.Classifier service 460 includes data curation module 461,machine-learning classifier module 462, output module 463, and trainingdata database 464. The modules depicted in FIG. 4 may be executed by aprocessor of a server or device hosting image analysis service 460. Theexecution of the modules may be driven by computer-readable instructionson a non-transitory computer-readable medium that, when executed, causethe processor to perform the operations described with respect to FIG.4. The databases depicted in FIG. 4 may be local to a device or serverhosting classifier 460, or may be remote to such a device or server, andaccessible by way of a network, such as a local area network or theInternet.

Data curation module 461 collects data used as inputs bymachine-learning classifier module 462 and feeds that data as an inputto machine-learning classifier module 462. In an embodiment, as patienthistory, current patient state, image biomarkers, and/or acousticbiomarkers are received, data curation module 461 feeds this data intomachine-learning classifier module 462 as input features. In anotherembodiment, such data is collected until a sufficient amount of data forinput into machine-learning classifier module 462 is received, and thenis input together to machine-learning classifier module 462.

Machine-learning classifier module 462 executes a machine learning modeltrained using data in training data database 464. In one embodiment, themachine learning model is a recurrent neural network (RNN) model, andtraining data in training data database 464 is updated on an ongoingbasis (e.g., for subsequent points in time) to improve the diagnosticaccuracy of the machine learning model. In an embodiment,machine-learning classifier module selects one or more machine learningmodels from several machine learning models to use depending on variousfactors. These factors may include which disease biomarker detectionand/or segmentation is being evaluated, and which anatomylocation/segmentation is being evaluated. For example, differentanatomical structures in the ear may each have dedicated machinelearning models trained using training data from those anatomicalstructures in training data database 464, leading to more preciseoutputs from the machine learning models. Similarly, different types ofbiomarkers (e.g., fissures versus blood flow markers) may each havededicated machine learning models for similar reasons. Such machinelearning models do not have to be (statistically) independent and may infact be partially overlapping in their combined solution space. Wheremultiple machine learning models are selected by machine-learningclassifier module, output form one module may form input of anothermodule. Further discussion of machine models, such as neural networks,and selection of the best machine models from a pool of candidatemachine models, is discussed in further detail in commonly owned U.S.Pat. No. 10,115,194, issued Oct. 30, 2018, the disclosure of which ishereby incorporated by reference herein in its entirety.

An example of machine-learning classifier module 462 classifying inputsmay be as follows. To detect tympanic membrane bulging, image analysisservice 130 may extract image regions of a relevant image and send themto machine-learning classifier module 462. Machine-learning classifiermodule 462 may select one or more non-orthogonal machine learningmodel(s) trained to detect a degree of bulging considering the region inand around the tympanic membrane. Machine-learning classifier module462, based on the degree of bulging, classifies the image as including aparticular disease. The disease may be determined based on a machinelearning model as well (e.g., with training data in training datadatabase 464). The disease may alternatively be determined based on aheuristic, or a combination of a machine learning model and a heuristic.The disease determination may include a probability that the diseasedetermination is accurate, which may also be computed based on themachine learning model, the heuristic, or a combination of the two.

In an embodiment, synthesized data for a patient (e.g., as synthesizedby data curation module 461) may be stored as a training example for apatient, the training example including the synthesized data that wasused as an input into the model selected by the machine-learningclassifier module 462, as well as a label that indicates whether themodel determined the patient to have an ear infection. The trainingdata, in the aggregate for several patients, may be stored to trainingdata database 464, and may be used to train or further refine one ormore diagnostic models. During training, parameters of the model may beupdated to improve an objective performance threshold, which is athreshold of outputting a correct diagnostic. Training may be concludedafter the objective performance threshold satisfies a condition (e.g.,being sufficiently high, or reaching a mark preset by an administrator).The updated parameters for the diagnostic model may then be stored inmemory of classifier service 460 for use when processing furthersynthesized input data.

FIG. 5 is an illustrative diagram of a therapy determination serviceused to autonomously generate a therapy for output to a user. Therapydetermination service 570 includes therapy determination module 571,therapy output module 572, and therapeutic parameters database 573. Themodules depicted in FIG. 5 may be executed by a processor of a server ordevice hosting therapy determination service 570. The execution of themodules may be driven by computer-readable instructions on anon-transitory computer-readable medium that, when executed, cause theprocessor to perform the operations described with respect to FIG. 5.The databases depicted in FIG. 5 may be local to a device or serverhosting therapy determination service 570, or may be remote to such adevice or server, and accessible by way of a network, such as a localarea network or the Internet.

Therapy determination module 571 receives the probability-baseddiagnosis from classifier service 160, and consults a therapeuticparameters database 573 to determine a therapy to recommend to anoperator. The therapeutic parameters database 573 may contain heuristicsdirectly from clinical practice guidelines and may be subject to changedepending on changes to the clinical practice guidelines. Therapeuticparameters database 573 may be automatically updated based on theservice detecting an update to clinical practice guidelines, or may beupdated by an administrator based on such updates. As an example, if thedetected disease is severe AOM with a 100% probability of being acorrect diagnosis, therapy determination module 571 may determine theappropriate therapy to be, based on entries of therapeutic parametersdatabase 573, which may include the type and dosage of antibiotics. Inone example, antibiotics are prescribed for children six months or older(e.g., as determined based on data from patient file server 110). Asanother example, if the detected disease is non-severe AOM with a 100%probability of being a correct diagnosis, therapy determination module571 may determine the appropriate therapy to be antibiotics orobservation for children 6-23 months old. Dosage calculations aredetermined based on patient weight and/or age, as dictated in theentries of therapeutic parameters database 573. In an embodiment, atherapy decision is determined on the basis of presence and/or severityof otitis media and/or the presence of otitis media with effusion,and/or the presence of serious effusion. In an embodiment, thetherapeutic parameters database 573 may contain a machine learning modelused to calculate the output of the therapy determination service 570with or without heuristics.

Therapy output module 572 outputs the determined therapy to the operatorand/or to the patient. The means of output may be through a display,speaker, haptic feedback, or any other means of communicatinginformation to the operator and/or patient. The output of the therapymay be through image capture apparatus 120, or through anotherapparatus, such as a client device of the operator or patient (e.g., amobile device).

SUMMARY

The foregoing description of the embodiments of the invention has beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the invention to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations are possible in light of the abovedisclosure.

Some portions of this description describe the embodiments of theinvention in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules, withoutloss of generality. The described operations and their associatedmodules may be embodied in software, firmware, hardware, or anycombinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments of the invention may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, and/or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be stored in a non-transitory, tangible computer readable storagemedium, or any type of media suitable for storing electronicinstructions, which may be coupled to a computer system bus.Furthermore, any computing systems referred to in the specification mayinclude a single processor or may be architectures employing multipleprocessor designs for increased computing capability.

Embodiments of the invention may also relate to a product that isproduced by a computing process described herein. Such a product maycomprise information resulting from a computing process, where theinformation is stored on a non-transitory, tangible computer readablestorage medium and may include any embodiment of a computer programproduct or other data combination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the invention be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsof the invention is intended to be illustrative, but not limiting, ofthe scope of the invention, which is set forth in the following claims.

What is claimed is:
 1. A method for diagnosing an ear infection, themethod comprising: receiving patient data about a patient, the patientdata comprising at least one of: patient history from medical recordsfor the patient, one or more vitals measurements of the patient, andanswers from the patient about the patient's condition; receiving a setof biomarker features extracted from measurement data taken from an earof the patient; synthesizing the patient data and the biomarker featuresinto input data; applying the synthesized input data to a traineddiagnostic model, the diagnostic model comprising a machine learningmodel configured to output a probability-based diagnosis of an earinfection from the synthesized input data; outputting the determineddiagnosis from the diagnostic model.
 2. The method of claim 1, whereinreceiving the set of biomarker features comprises: obtaining an image ofa portion of the patient's ear; extracting one or more biomarkerfeatures from the image, the extracted biomarker features including alikelihood of whether the sample contains an ear image object and alocation of the ear image object.
 3. The method of claim 2, wherein theimage is at least one of a two-dimensional image or a three-dimensionaloptical coherence tomography image.
 4. The method of claim 2, whereinthe portion of the patient's ear comprises at least one of a tympanicmembrane, an anatomical structure adjacent to a tympanic membrane, anear canal adjacent to the tympanic membrane, a malleus, an umbo, and alight reflex.
 5. The method of claim 2, wherein extracting the one ormore biomarker features from the image comprises: obtaining a set ofsamples of the ear image, each sample corresponding to a location in theear; and for each of the set of samples, applying the sample to atrained feature detection model, the feature detection model comprisinga neural network that is configured to output a likelihood of whetherthe sample contains an ear image
 6. The method of claim 2, wherein theimage was re-taken in response to a determination that a prior imageincluded an abnormality that would interfere with disease assessment. 7.The method of claim 1, wherein receiving the set of biomarker featurescomprises: applying a pressure stimulus to inside an ear of the patient;receiving an acoustic response from the applied pressure waves;extracting acoustic biomarker features from the received acousticresponse of the ear to a pressure stimulus; and synthesizing, into theinput data, the acoustic biomarker features.
 8. The method of claim 7,wherein the pressure stimulus is applied using at least one of pneumaticotoscopy or tympanometry.
 9. The method of claim 1, wherein the imagebiomarker features each further indicate a confidence value thatreflects a confidence that the indicated anatomical feature of the earwas accurately determined.
 10. The method of claim 1, furthercomprising: determining a therapy for treatment of the ear infectionbased on the determined diagnosis; and providing a description of thetherapy.
 11. The method of claim 10, wherein determining the therapycomprises: accessing a therapeutic parameters database; identifying anentry of the therapeutic parameters database that corresponds to boththe determination of whether the patient has the disease, as well as theprobability that the determination that the patient has a disease istrue; and determining the therapy to be a therapy indicated by the entrybased on the determined diagnosis.
 12. The method of claim 10, whereindetermining the therapy comprises: applying a machine learning modelpreviously trained to associate the determined diagnosis and the patientdata with a therapy.
 13. A computer program product for diagnosing anear infection, the computer program product comprising acomputer-readable storage medium containing computer program code for:receiving patient data about a patient, the patient data comprising atleast one of: patient history from medical records for the patient, oneor more vitals measurements of the patient, and answers from the patientabout the patient's condition; receiving a set of biomarker featuresextracted from measurement data taken from an ear of the patient;synthesizing the patient data and the biomarker features into inputdata; applying the synthesized input data to a trained diagnostic model,the diagnostic model comprising a machine learning model configured tooutput a probability-based diagnosis of an ear infection from thesynthesized input data; outputting the determined diagnosis from thediagnostic model.
 14. The computer program product method of claim 13,wherein receiving the set of biomarker features comprises: obtaining animage of a portion of the patient's ear; extracting one or morebiomarker features from the image, the extracted biomarker featuresincluding a likelihood of whether the sample contains an ear imageobject and a location of the ear image object.
 15. The computer programproduct of claim 14, wherein the image is at least one of atwo-dimensional image or a three-dimensional optical coherencetomography image.
 16. The computer program product of claim 14, whereinthe portion of the patient's ear comprises at least one of a tympanicmembrane, an anatomical structure adjacent to a tympanic membrane, anear canal adjacent to the tympanic membrane, a malleus, an umbo, and alight reflex.
 17. The computer program product of claim 14, whereinextracting the one or more biomarker features from the image comprises:obtaining a set of samples of the ear image, each sample correspondingto a location in the ear; and for each of the set of samples, applyingthe sample to a trained feature detection model, the feature detectionmodel comprising one or more neural networks configured to output alikelihood of whether the sample contains an ear image
 18. The computerprogram product of claim 14, wherein the image was re-taken in responseto a determination that a prior image included an abnormality that wouldinterfere with disease assessment.
 19. The method of claim 13, whereinreceiving the set of biomarker features comprises: applying a pressurestimulus to inside an ear of the patient; receiving an acoustic responsefrom the applied pressure waves; extracting acoustic biomarker featuresfrom the received acoustic response of the ear to a pressure stimulus;and synthesizing, into the input data, the acoustic biomarker features.20. The computer program product of claim 19, wherein the pressurestimulus is applied using at least one of pneumatic otoscopy ortympanometry.
 21. A diagnostic product for diagnosing an ear infection,wherein the diagnostic product is stored on a non-transitory computerreadable medium and is manufactured by a process comprising: for each ofa plurality of patients: receiving patient data about the patient, thepatient data comprising at least one of: patient history from medicalrecords for the patient, one or more vitals measurements of the patient,and answers from the patient about the patient's condition, receiving aset of biomarker features extracted from measurement data taken from anear of the patient, synthesizing the patient data and the biomarkerfeatures into input data for the patient, and storing a training examplefor the patient, the training example comprising the input data for thepatient and a label that indicates whether the patient has an earinfection; for a diagnostic model, the diagnostic model comprising amachine learning model that is configured to output a diagnosis of anear infection: training the diagnostic model by repeatedly applying atraining example from the plurality of training examples to thediagnostic model and updating parameters of the diagnostic model toimprove an objective performance threshold, and stopping the trainingafter the objective performance threshold satisfies a condition; andstoring the updated parameters for the diagnostic model on the computerreadable storage medium.
 22. The diagnostic product of claim 21, whereinreceiving the set of biomarker features comprises: obtaining an image ofa portion of the patient's ear; extracting one or more biomarkerfeatures from the image, the extracted biomarker features including alikelihood of whether the sample contains an ear image object and alocation of the ear image object.
 23. The diagnostic product of claim22, wherein the image is at least one of a two-dimensional image or athree-dimensional optical coherence tomography image.
 24. The diagnosticproduct of claim 22, wherein the portion of the patient's ear comprisesat least one of a tympanic membrane, an anatomical structure adjacent toa tympanic membrane, an ear canal adjacent to the tympanic membrane, amalleus, an umbo, and a light reflex.
 25. The diagnostic product ofclaim 22, wherein extracting the one or more biomarker features from theimage comprises: obtaining a set of samples of the ear image, eachsample corresponding to a location in the ear; and for each of the setof samples, applying the sample to a trained feature detection model,the feature detection model comprising a neural network that isconfigured to output a likelihood of whether the sample contains an earimage
 26. The diagnostic product of claim 22, wherein the image wasre-taken in response to a determination that a prior image included anabnormality that would interfere with disease assessment.
 27. Thediagnostic product of claim 21, wherein receiving the set of biomarkerfeatures comprises: applying a pressure stimulus to inside an ear of thepatient; receiving an acoustic response from the applied pressure waves;extracting acoustic biomarker features from the received acousticresponse of the ear to a pressure stimulus; and synthesizing, into theinput data, the acoustic biomarker features.
 28. The diagnostic productof claim 27, wherein the pressure stimulus is applied using at least oneof pneumatic otoscopy or tympanometry.
 29. The diagnostic product ofclaim 21, wherein the image biomarker features each further indicate aconfidence value that reflects a confidence that the indicatedanatomical feature of the ear was accurately determined.